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tXECUTIVSAfl>L\Ig. 

This  stucly  is  an  examimtion  of ' the  tble  of  the  U*  S*  Amy  Operational 
Test  :^d  Evaluation  Agency  (OTEA)  in-  accomplishing  operational  testing  and 
evaluation  of  major  systems  in  the  Ar^  as  directed  hy  DODD  5000.3#.  ‘^Test 
and  Evaluation//'  dated  19'‘lanuaiy  1973. 

Ihe  study  vtraces  the.  evolution  of  operatiOTal  testing  and  evali^tion 
for-inajor  systems  in  the  Aimiy  begiimi^  with  the  findings  and  recommenda- 
tiofuv  of  the  Blue  Ribbon  Defense  Panel  Report  in  duly  197Q.  Vai*iou|rOSD 
and  DA  raemorandunis  provided  the  bc(^|rpund  for-  .the  events,  leading  up  to 
the  establishment  of  an  independent-^st  agen^  in  the*  i^my  for  the  conduct 
of  operational  testing  and  evdluatidn. 

iDODD  5000i|#  "Test  and=  EvalViatiOT#"  and  the  DA  "Letter  of  Ihw  t:mctions 
(LOXOrfor  IraplMcnting  the>  New  Materiel  Acqiiisitioh  Guidelihes,"  dated 
23  Mgust  l97?f  .provide  -the  basis  fOT  OTEA'  s cji^ter  wi1h*  respect  tp.  its 
i^spMsitilities  for  the  planning  and  conduct  of  operational  i-esting  dhd 
e^uaticn  (OT^).  OTEA  in  the  accOTplishment  p?  its  0T^iE  Mssion-  in.tsr~ 
faced  with  the  materiel  deyeloper#  Pfbgram^  K^  user#,  combat  developer, 

trainpr,  and  the  DA  Staff f The  resets  of  its  independent  evaluati^  of 
opeTOtional  test  reports  for  major  systems  are  /provided  directly  to  the 
j\my '®ief  of  Staff  and  subsequently  to  raembcr§=  of  the  AGARC  and  DSi\^ 
for  use  in  the  decision  making  process. 

'^ile  OTEA. has  not  been  in  e:3Lstence  long  enough  to  effoctively 
assess  its  value  in  irapro^^g  the  materiel  acq^sitibn  process,  it  is 
safe  to  say  that  the  jVrmy  has  estexblishod  the  noceahaiy  C^owork  ai^ 
organiaation  tp  objectively  accompligh  OTfiS  for'inajor  systeirs#  Thewfore^ 

ii 


wli*  the  estaWJ^^ent  of  0!iM  as=  ^ agenpy  reporting 

directly  to  the  Afttf  Chief  of  Staff,  the  Argy  has:  essentially  complied 
vdl*  SCO  poli<y  conduct  of  OjCi^  for  major  pystems. 

Some  conflict  botueon  the  sei^ce  use  portion  of  development  testing 
>'(DX)  conducted  tK  tte  U,  S,  Army  fest  and  Ev?14^tJon  Conaand  and  operation- 
S1  testing  (OT)-conuuctod  by  OTEA  and  the  user  e^dots.  The  service  Use 
;^rti<m  of  DT  is  operational  iu  rmt^  and  sho>dd' he  eliminated  from  DT, 
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Genera 

Operational  Test  and  Evaluation  (0^^)  was,  until  sul^ssion  of  the 
‘Blue  Ribbon  Defense  Panel  Report,  an  area  of  "the  oateriel  a^uisition 
process  that  received  soaething  less  than  adequate  attentlra  within  the 
=^partaent  of  Defehre*  Generally,  qperStitmal  units  had  little  to  do 

a ^stea  until  after  it  was  in  pjrotoction*  Thus,  operetional  de- 
ficiencies discovex^  after  a najor  sys^  was  in  production  required 
rether  substantia  resources  to  correct*.  In  Bost  cases^  -Uie  results  of 
OTSS  were  not  generet^  ®arly  enou^  lapac.t  on  the  iap^^tant  production 
decisions  tdxich  l^  i» 

Althou^  the  had  recogniz^  tte  need  for  e^lierHOT^  prior  to 
receiving  the  Blue  ^bbon  Defense  Panel  Bepoi't,  changes  t§  qrstea  1 

rere  slow  in  being  inpleiaented* 

At  the  tine  of  ^bmission  of  the.  ^ue  Ribbon  Defense  Panel  Report  in 
July  1970,  0TS£  for  Sajor  Arcy  systeas  ;^s  conducted  ly  the  D,  3*  i\rcy 
Test  and  Evaluation  CoRBnand  (TECGE-I)^  a ^bordinate  coDOond' of  the  U*  S*  | 

i 

/aiy  Materiel  CooraaM,  the  developer*  1SC®I  perforned  bo^  Enginrerlng  * 

Tests  and  Service  Tests  on  Any  aateri^*  The  Engineering  Tests  were 

^rfonaod  by  ISOT4  at  a nuaber  of  provi^  grounds,  ihile  Service  Tests 

were  pe^fPoraed  by  ^ of  the  six  Service  Test  Boards  which  rere  subordinate 

to  TSCOE-I,  These  Sex^cc  Tests  did  include  sone  operation^  testing.  The 

Service  Boards  were  collocated  with  the  user  and  did  attempt  to  represent 

■Uie  user  point  of  view*  However,  the  Blue  Ribbon  Defense  Panel  found 


1 


that  for  OTcS  to  bo  effective,  it  ^ould  be  conducted  by  ii^ependeht 
0T£E  organi^tlons  tdiich  reported  direci^  to  the  ®ief s of  Service 
(6;?)> 

As  a result  of  the  Blue  Ribbwi  Defense  Panel  Beport  and  guidance 
provided  by  the  Office  of  the  Secretary  of  Defense,;  the  Secretary  of  the 
i\rcy  on  22  Septenber  1972  directed  activation  of  the  U,  S.  Aroy  Operational 
Test  and  Svalxiation  Agen^r  (OSSA),  0T3A  was  estabdi^ed  as  a field 
operating  agency  under  the  staff  supervision  of  the  ACS^R  to  act  as 
the  Arcy's  independent  operational  test  arai  evaluation  agency.  The 
decision  to  establish  0T3A  dOLninated  a long  exchange  of  nebpranda  between 
the  ifepartnent  of  Defense  and  the  Departaont  of  the  Aray  (12:2), 

Pur^se  of  ^thc  Stugy 

The  pufjwse  of  this  study  is  to  e:<agine  the  role  of  the  U*  5.  Arqy 
Operational  Test  aad=  Bvaluation  Agency  (OISA)  In  ixjjieaenting  the  guidance 
contained  in  DOD  Directive  5000,3,  Tost  and  Evaluation,  with  respect  to 
OTdS  for  nalor  ^stras  in  the  Arqy,  Additionally,  the  study  attempts  to 
exanine  the  interface  between  0T2A,  the  user,  the  developery'^roject 
Manager,  and  the  Arcy  Staff  in  accoa^ishing  the  operational  test  and 
evaluation  cdssion. 

Definitions 

Developeent  Test  and  Svaluation  (DK3),  Technical  requiraaents- 
orient^  testing  conducted  threw^out  the  develppcent  cycle  to 
deteraine  the  degree  to  which  a lystea  sects  perfonasne®  specifica- 
tions aM  to  aeasure  operability  (including  Reliability^  Avail- 
abHity,  Maintataabllity  (RAll)f  coapatl^i^*,  interopcrebility, 
panHaaebine  interface,  safety  a^  logistical  c^siderations)  fOr 
a pros^ctive  u^r.  Earlier  testing  assesses  technical  risics. 
Subsequent  DT£S  resolves  or  ninj^zes  design  ri^  probidas  ai^ 
provides  an  esttoation  of  tho  systea»s  sllitary  utility  (3s  2), 


Operational  Test  and  Svnluation  (OTSS),  0T2S  is  testing  and 
e'^uation  o£  a ^stea  to  detemine;  its  nilitaiy  utility,, 
operational  effccti'/cness,  and  operatiraal  suitability 
(including  Reliability,  Availability,  Ifeintainability  (RAiO, 
compatibility,  interoperability,  logistit^and  training  reqi^e- 
Qcnts);  the  new  cystcn’s  desixtibillly  from  the  user’s  viewpoint 
considering  available  cquipacnt  and  the  nev  systen’s  benefits 
or  burdens;  the  need  for  any  nodifications;  and  the  adeqt^.^' 
of  organization,  doctrine,  operating  techniques,  and  tactics 
for  its  eaDloyseht  and  the  jystem  for  its  naintenance  support 
(2:1). 

rlajor  S^stery^rpgras.  Kajor  programs  are  those  as  designated 
by  the  Secretary  of  Defcnse/lteputy  Secretary  of  Defense,  This 
designation  ^all  consider  (l)  dollar  value  (programs  vith  ah 
estinated  HDTcS  cost  in  excess  of  50  nillion  dollars,  or  an 
estinatod  production  cost  in  excess  of  >500  niUiwi  dollars);;: 

(2)  national  ur^n^-;  (3)  reconnerKiatiqns  of  DOD  component 
heads  or  02)  officials  (5:1). 


Kathode 


TV  ana  ocone 


The  nethodoiogy  us^  in  this  study  iaduded  library  resear^^  intervieus 
with  personnel  assi^ed  to  OELl  and  the  Staff,  ai^  inf oraal  discus- 
sions with  students  of  TKC  (Hass  74-1.  In  addition,  visiting 
Kanagers  were  quostiaicd  about  their  relattonships  with  OTSa. 

Threughout  the  study  caphasis  is  plac^  upon  ihe  Blue  Ribbra  Defense 
Panel  Report,  02)sH^oranduns,  DOD  Direc^yes,  and  various  Ar^  documents 
in  tracing  the  steps  leading  up  to  the  establishaent  of  03SA  am  its 
rrde  and  responsibi^ties  in  acccaplishing  the  0T£S  nissioo. 


This  study  will  basically  follow  a descriptive  analysis  a^rmch. 
Since  publication  of  the  Blue  Ribbon  Dafense  Panel  Study,  OSD  tas  jlac^ 

c<mtimied  m^asis  upon  the  estabUs^rsent  of  an  ii^epeMent  opsmtional 

% 

test  and  evaluati<m  agency  within  eat*  of  fee  Services  aM  fee  cornet 
of  timely  and  objecUvo  0T2S, 
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Chapter  II  is  a historical  review  of  the  events  leadijig  up  to  the 
estallishment  of  OTEA  hy  the  Army,  In  Chapter  III  discussion  centers  on 
OTEA  and  its  responsibilities  and  roles  for  insuring  adequate  OT&E  for 
major  systems.  Chapter  IV  addresses  the  time  phased  accomplishment  of 
OT&E  functions  for  major  systems  and  some  of  the  recommendations  of  the 
recent  AltlRC  study.  Chapter  V presents  a brief  summary  and  conclusions 
of  the  study. 

Due  to  the  limited  time  availaHe  to  conduct  this  study  and  the 
fact  that  OTSiS  vdthin  the  Army  is  still  in  soraewha.t  of  a transition  stage, 
no  attempt  was  made  to  examine  actual  OTfiiE  documentation  on  individual 
systems.  Furthermore,  no  assessment  vras  attempted- regarding  OISA’s 
performance  in  tl\e  actual  conduct  of  OTSiE  for  ma j or  systems  and  the 
adequacy  of  OTSiS  data  provided  to  support  AS/iRC/DSARC  decisions. 


CILiPTSR  II 
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IIISTORIC/iL  REVIEW  OF  EVENTS  LE/iDIKG  UP  TO  THE  ARIS’S 
ESTABLISHi-ISI^T  OF  OTEA 


Blue  Ribbon  Defense  Panel 

In  its  report  to  the  President  and  the  Secretarj''  of  Defense  in  July 
1970,  the  Blue  Ribbon  Defense  Panel  stated  that: 

"Operational  test  and  evaluation  has  been  too  irifrequent, 
poorly  designed  and  executed,  and  generally  inadequate"  (7:2), 

Tlie  Blue  Ribbon  Defense  Panel  uas  appointed  ly  Melvin  R.  Laird,  Secretary 

of  Defense,  on  3.9  Jiily  1969,  to  study  the  entire  organization,  structure, 

and  opera.tion  of  the  Department  of  Defense,  The  panel's  report  v;as 

•basically  composed  of  a sitomary  volume  which  contained  113  recommendations, 

a brief  rat.iohalc  for  each,  and  I4  separately  publivShed'  appendices. 

Appendix  F,  to  Blue  Ribbon  Panel  Report  was  devoted  specifically  to 

operational  test  ?-nd  evaluation  (OTfS), 

The  Blixe  Ribbon  Panel  Report  concluded  that  the  results  of  0T£E 

boi:ig  conducted  at  that  time  x/ore  not  being  used  l^’',  nor  in  many  cases 

made  availatxLe  to  those  Department  of  Defense  agencies  which  needed  the 

information.  The  panel  also  found  that  OTfiS  was  not  adequately  managed 

or  supervised  at  the  Office  of  the  Secrotaiy  of  Defense  level,  IVirtlief, 

the  panel  found'  that  OT^  v/as  best  accomplished  vAien  it  was  done  by  an 

organization  independent  of  both  the  developer  and  the  user,  reporting 

directly  to  th^  Cliief  of  the  Service  (?:7), 

Evolution  of  OT&E  Jinco  1970  Within  tho  i\rmy 

■ifter  ravievdng  the  recommendations  of  the  Blue  Ribbon  Defense  Panel 
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in  July  1970,  tho  Socrotary  of  Defense  expressed  his  concerns  oh  0Tt*E 


t 
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;to  +He  Secrotapies  oi*  ‘fchf>  thr^e  fervieea*  XjS;  a JS5^t  reply  fo  the 
Secretary  of  Defense  on  "'2.  Je*  »Jb^r  397Q,  Service  Secreferie  si 
pledged  their  all  in  reblsiag  the  conduct  of  "0T2S;!(9:l')’« 

tittle  was  changed  './Ithln  the  Oeplrtmont  of  Amy  in  1970*  The  Art^ 
was  ?in  the  process  of'?.  m;jor  reorrenisatien  and;  revisions  for  OT^  were; 
merely  part  of  ^^e  overoJLl ' problems  to»  bc  solved#  The  main  ^obl^- witHi 
the  i\ray '^stoni  a.'-as  that  the  dov<^oper’ tested  and' evaluated  "^e  opera- 
tional suitability  of  the  ^sterns  being  developed, 

iOn  11-  Febmary  19^,  Mr.  Havid  Packard'j;.  Depu%  Secretaiy  of  Defensej; 
stated  that  the  'dptlsm  mettod  for  conducting  operational'  te|t  andi 
evaluation* -coidaUbest  ibe  performed  by  an  agency  vMch  1|>  separate  and 
distinct  Ifpm  -^e  deyei^^  co^and  and^ihlOh.  report s-'dire^^^  to;  the 
'^ief  of  the  Serylcei,  Fur.^er, 

» i #. ‘vdthjn  the’ Service  headquarters  ;|taff  i/%ere\nee.d|: 
sbe  ani  off lc|  idtH:  c3.oaf  OT^-  idenfific|tion  ^to  proyidp  staff  assisr- 
tahc.e);dird,ctly  to;  the-  ^rvi'ce  Chilf  an®  to  provide'  a he^quafjers 
foc£d*  :.point  for  %e  independent  ^^^at&nal  test  and  ey^uatlOh 
iiol®  agency  ,(l6:i) 

The  Deputy  Seerfta:^  of  Defense  memo  ;c]^hcided  wi&  and:  reinforced' 
the ‘pyeraH'  findings  Of  the=  Blue  Ribbon  Dofehse  Panel  Report;,  iloweyer,. 
v;hefc  tlie  jpaneltfs  report  recommended,  IJr,  Packard  la  raemO  dif^eptod  tha,t 
ajppr^ria^  actiph  be-  taken#  This  memorandum  also-  annc^cedl  (toe  eftallito- 
meht-‘pf  a-'Deputy  Director  for  Test  and'lilvaluatiphi’Vdthito  ODDI^ 
acrpssrthfrhpard-  respphsibfiitiosrfor  tOSD  in)  tost  and  equation  matterf- 
(10:1)# 

(The  igmy  a|;  a plft  of  its  massive;  reorganizutt^  of  1970  had 

initiated  a detailed ;reviod'of  toe  totod  test  and  bvnlMtioh'iprog^ra# 
Irapllmentdtion  of  a new  organization  and  test  phiipsoply  was*  to  ;bc 
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initiated  ly  a rewrite  of  AR  70-iO»  ”Test  tjmd  Evaluation  Durii^;  Develop-  | 

. ^ - I 

raent  and  Acquisition  of  Materiel,"  | 

i 

Deai^atlon  of  USACDC  as  Armyt-s  OTitB  Actj^tv  j 

■Pendinp  revision  of  /iR  70-10,  interiin  policies  and,  instructions  for  ! 

t 

operational  test  and  ovaltiatioh  in  support  of  the  m^riel  acquisition  I 

process  were  issued  by  DA  lotter  dated  3Q>  November  19?i*  The  Assistant  i 

i 

Chief  of  Staff  for  Force  Dcvoldpnient  ’,7a s (designated  as  the  /irray  Staff  | 

focal  point  for  all  OTSiS  matters  and  ttle  -U,  S,  /irny  ■Combat  Developments  | 

Comra^d  (Ua\CpC)Hwas  desi^ated  as  tlie  A^f*  s 0T£E  >activity,  separate  I 

and  distinct  frpra  ^e  develdper  j 

.in  its  role»  as  the  Ain^-irs-^OT^  acti^ty,  USACDCmV^s  to  be  rf^nsible  j 

for  reviewing  andl  approving  “^e  test  plan' for  the  Bonded  i^ryide  'Test  | 

(EOT))  to  be  cpnd^toi^^  TSCQ-I^  In  additipn,  USACpCA>i8  to  par  | 

in  tost  and- fgabmiti  its  iMependcnt -ev^uation  of  i^e  | 

operation^  effect-ivenes's  ard^suitabilily  directly  tofllQ  DA  (AC^R),  f 

Thdfesults  of  EJT‘  would  pr^de  limitedi^  test  data  upon  which  | 

h 

to=  h^9  a lliit^' productiw<decision  i^jpr  itws;  for  furti^  \ 

' - > 

opordtional  -testing,  USACDd^iwould  then  (develop  a plm  for  an  Intensified  j 

Confi^tpry  Troop  Test  (ICTT)  to  be  exerted  by  Cp^RC  (pr  other 

dei^ated  major  icommand)  wi^  assistance  from  USAffiC,.  This  user  field 

test  iwas  to  bo;  conducted  in  'ds  realisttcifan  operatl^ai  environment  as 

possible#  USACDC  would  thenMsUbrait  the  ICTT  report  along  with  its 

independent  evaluation  of  i^e  igystem * s durational  effectiveness  and 

suitebility,  to  i^  DA  (ACI^r)  for  review  prior  tP^^e  OSD  major  (production; 

decision  (DS/dlC  illi,  ^is  -raport  in  would  forwarded  to  OS^ 

for  its  review  prior  to  DSi\M  III  (1:3)  s 
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liJhile  designatiiri -or  U&\®C  a^  pT^  acti-^ 

in  the  right  direction^  it  fa^etd-  to  fuUy  meet  to®  S^i^i^^ce 

provided  hy  Offl,  Additton^y^  ih  'i^2,  the  Aitiy  had  erabarkei  on^ 
extensive  internal  reorganization  Atudy  v^ich  envisiohed;.abolishceht^^o^ 

USACDC,  Thus  on  22  September  1972j  "ttie  fepretary  of  the  Arny  ■directed? 
activation  of  the  U,  S,  iVrny  Operatiohal  Test  and  E\^uation=  Agency 
(0T3A)  as  the  ilrmy's  independent  agency  responsible  for  OTSS  :{i2;-i) , 
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C11(\PSER  111 

U.  S.  iUB-K  OPEMTiOKAL  TEST  AIJD 
]3Vj\LUATIOK  AGEIICT  (O'JEA) 

OTEA  VJS3;  activated  on  22  September  197?  at  the  direction  of  the 
Sccwtaiy  of  the  i\rny*  OTEA  vkis  cstabli^ed  as  a field  pirating  ageri^ 
Tiiider  the  Staff  aupei*vision  of  the  AC3F0R  and  located  at  Port  Belvoir, 
Virginia,.  The  agency  is  commanded  by  a Major  General,  who  also  currently 
serves  as:  the  Dopuly  ACSPOR  for  Test  and  Eveluatibn  (12:1)  * 

OTSA’ s initial  uuthbr3,aed  strength  was  120,  Curreniiy  the  organiza% 
tion  is  au-ttbrizod  12Ji.  officers,  24  EH,  and  102  civilians  for  a total  of 
250  perBpnhPl  (12:11) , 

Since  bstablishmoht  p'f  OTEA,  AR  7C--10j  “Test  and  Evnlmtion  During 
Development  and  Acqulsitipiv  of  Materiel,"  and  related  regulations  have 
been  \mder  revision  and  have  not  been  officially  published,  Therofo.re^ 

OTSA  derived  much  of  its  pidance  and  responsibilities,  fr^  tie  HO  DA 
23  August  1972^  "Letter'  of  inotnxetions  (LOIj  for  Implementing  the  New 
Materiel  Acquisition  Guidelines,"  Although  this  LOI  was.  published  one 
month  prior  to  the  activation  of  OTEA,  it  hevertlioloss  addressed  the 
responsibilities  of  OTEA  ajpd  stated  tliat  OEA  will  be  rbs^’nsible  for 
assuring  that  adequate  operational  testing  (OT)  is  accomplished  for 
,maj or  ^sterns  (l3sH3)  • 

The  LOi  stated  that  in  the  case  of  major  ^sterns  O^A  will  actively 
participa’tp  with  the  dosighatod  user  in  'Uio  planning  for  and  conduct  of 
operational  testing,  and  wdJ.1  prepare  an  indepondent  evaluation  of  the 
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adequacy  of  the  testing  and,  the  validily  of  the  results  upon  completion  of 
each  phase  of  OT,  Involvenent  hy  the  OTSil  in  QT  of  non-major  systems 
may  be  directed  ty  IIQ  DA  on  a case-lyr-case  basis  (l3fU3)  • 

flae  LOI  ciso  states  that  OTEA  is  responsible  for  preparation  of  the 


OT  portion  of  the  Coordinated  Test  Program 


for  major  systems,  OTEA 


must  coordinate  ^dth  the  developer  and  harmonize  the  OT  portion  of  the 
CTP  vdth  the  Development  Test  (DT)  portion,  using  DT  as  the  baseline, 

OTEA  must  determine  when,,  vihere,  how  and  by  wh<®  OT  ’ydll  be  accomplished; 
coordinate  Adth  tlie  user  and  determine  Ahat  user  support, must  be  provided 
to  accOTiplish  Of;  prepoa’c  the  OT  potion  of  the  CTP  and  provide  it  to  the 
materiel  develo;^r  (I3sr4)'» 

The  materiel  developer  (Program  Manager  if  designated)  determines 
whatssupport  must  be  provided  to  accomplish  the  entire  tost  program; 
obtains  input  the  trainer,  conl^t  developer  and  lo^stician;  prepares 
a CTO.;  coordinates  it  Adth  OTEA,  the  user,  the  trainer,  the  combat  develop- 
er, and  the  logistician;  and  subject  to  concurrence  of  QTBA,  publishes 
and  distributos  the  test  program.  If  later  events  require  any  change 
to  toe  OT  portion  of  the  -CTP,  OTEA  takes  toe  initiatiYe  in  making  toe 
changes,  again  harmonizing  them  Adth  DT,  and  pxrovides  them  to  toe  materiel 
dovetoperAro^ara  Manager  for  publicaticm  as  an  amendment  to  the  OT, 

The  materiel  dsvelopcrAj^oc*^  Manager  obtains  the  ccmcurrence  of  OTSA 
before  making  dianges  in  toe  DT  portion  of  the  CTP  which  may  impact  oh 
OT  (13:F4). 


Concurrent  with  toe  Army*  s preparation  of  its  LOI  .prevdously 
referred  to,  DOD  A^as  preparing  DOD  Directive  5000,3,  ”Tost  and  EA^mtion," 


ii.Yiliiiiiiii4lriMi*Niii^ 


This  diwctive  which  was  forraaliy  pahlishod  on  19  January  1973»  superseded 

the  previously  issued  Deputy  Secretyry  of  Defense  memos  vAiich  had  provided 

guidance  concerning  OTfi£, 

pOD  Directive  5000.3  provides  ttat: 

“In  each  DOD  coaponont  there  \dll  be  one  major  field  agency 
separate  and  distinct  from  the  dcveloping/procuring  command 
and  froa  the  using  command  ;*ich  v?ill  be  responsible  for 

and  lihich  will:  | 

A,  Report  tile  results  of  its  indbpendeat  test  and  evalv-a- 

tion  directly  to  the  Hilitaty  ^rvico  Qiiof,  or  Defense  Agency  ! 

Director, 

B,  Recoanend  dircctOy  to  its  Military  Service  f^iief  or  ; 

Defense  Agency  Dtyector  the  accooplishmcnt  of  adequate  OT^, 

G,  insure  that  the  0T2E  is  effectively  ^planned  and  con- 
ducted, in  addition,  each  DOD  Component  irlil  provide  vdtHin 
its  immediate  headquartors  staff  a full-time j strong,  focal 
point  organize. tlOT  to  assist  the  independent  OTiiR  field  agencj’- 
and  to  Iceop  its  Military  Service  Chief  or  Defense  Agency 

Director  fully  informed  as  to  the  needs  and=  accomplishments  ; ] 

(8:3).” 

Thus  at  tiiis  time,  0T3.\*s  diartcr  stems  from:  the  January'-  1^3  DOD 
Directive  on  Test  and  .Bvaluation  (5000,3)  and  the  Army’s  "Letter  of 
Instruction  (LOI)  for  Implementing  the  New  llntertcl  Guidelines"  published 
on  23  August  1972. 

o'i^v  oreanigaUm  ! 

I - 

In  order  to  acccaplish  its  testing  rosponsibilitles,  0TSi\  is  j 

f - 

organized  according  to  toot  functions  as  opposed  to  system  functional 
areas  such  as  artillery,  amor  or  infantry,  0IE.\  has  five  functional 
divtSiohs  lAilch  inclMe  a Tost  Design  Division,  Field  Test  Division, 

Evalutttion  Division,  Operation"!  Support  Divisiem  and  a Technical  Support 
Division,.  As  their  titles  imply,  these  divisions  are  responsible  for 
tost  design,  taJrin'j-  the  tost  design  to  the  field,  ixitting  on  the  tost 
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in  a sheeted  environnenty  prepdrihg  an  inde^ndent  evaluation  of  the 
test  reffj3'*'3  pjid  tiiroughout  tlic  process  coordinating  to  insure  that  aH 
support  required  for  the  test  is  avQ0ahIe  at  the  right  tine  and  places 
The  Technical  Support  Division  provides  statistical  design  and  analysis 
^pport  to  tlie  other  dif  cions  as  requixed  (I2:l0)* 

There  is  a coordinating  staff  vdiich  inci^udea  the  Test  Managers  jmd 
the  Plaj  .s  and  Operation  Division.  The  Test  Managers  provide  the  detailed 
nanageneat  of  the  OT^  progran  for  the  major  systems..  The  Plans  and 
Operations.  Division  is  rcsponsillc  for  the  overall  coordination  of  all= 

OTSA  programs.  This  division  is.  respons5.hLe  for  Amy  user  tost  poli<^, 
admi^stration  of  the  Tost  Schedule  itoview  Comitteo/i*ive  Year  Test 
Program  process,  for  programming  and  hudgetihg-  0!3'ii\  funds  for  0T3A»  s user 
tests  and  for  providing  the  functional  staff  vdth  throat<data  requiwd  for 
operational  testing  (12;11). 

Located  in  the  Pentagon  there  is  an  OTEA  Coordination  Office  which 
provides  an  immediate  point  of  contact  for  coordination  vdth  the  Arr^y 
Staff  and  OSD  in  OTJS  activities. 

Ghanfig.g  M W, 

Due  to  the  planned  abolishment  of  ACSPOIt  as  a separate  element  on 
the  Staff  in  the  near  future,  OTEA  will  report  directly  to  the 
Chief  of  Staff  vAien  ACSPOR  is  abolished.  This  change  is  programmed  to 
occur  on  20  May  1974.  In  many  respects  this  rfiould,  in  the  opinion  of 
the  author,  insure  a more  valid  and  objectiw  evaluation  of  OT  results. 


■'Ii'.i-  -I  iiuji«PLiiJiiji^w«y[f^ji))HI.,Uyiyj_!!i.  .1.1. 
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CIL\PT3R  IV 

ACCX^iPLISIIl-IEnT  OF  OT££ 

FUNCTIONS  iOR^lLiJOR  iVK-ff  SISmiS 

Participation jOT  Special  TggkJ'orco 

Upon  DA  approval  of  a Required  Operational  Capability  (ROC)  for  a 
tnajor  systen,;  a Special  Task  Forco  Is  forned  by  DA  to  develop  a recotnaended 
approach  to  f^fUl  tlie  systcni  need  described  In  the  approved  HOC,  OTSIA 
will  provide  wrkir.g  representetivos  to  this  task  force*  The  OTSA  re- 
presentativos  on  the  task  force  \dll  participate  in  the  preparation  of 
Section  7 of  the  draft  Development  Concept  Faj^r  (DGP),  ’.dilch  covers  test 
and  evaluation*  This  requires  a samnmry  of  the  plan  f or  OT^  f or  the 
new  system  and  will,  state  the  objectives  of  each  test  and  tte  critical 
is^ies  to  be  resolved  by  testing*  The  fin:J.  import  of  the  Task  Force 
contains  a plan  for  test  and  evaluation  vAiich  provides  the  basis  for  the 
CTP* 

During  this  phase  OTSLI  wdll  actively  interface  with  the  Project 
Manager  Designee,  the  designatyd  user,  materiel  developer,  combat 
developer,  trainer  and  the  DA  Staff  (13;E2)* 

Pregaratibti  .of  - the  Dovelonnent  Plan 

During  preparation  of  the  Development  P^,  :^cifically  the 
Coordinated  Test  Program  (CTP),  the  same  OTSA  personnel  Uho  worked  vdth= 
the  Task  Force  will  participate*  It  is  during  preparation  of  the  CTP 
that  OTSjI  determines  vdicn,  whenre,  how,  and  by  idiom  operational  testing 
will  bo  accooplislicd.  The  CTP  indudos  comprehensive  plans  for 
accomplishing  both  dovolopment  tests  (DT)  and  operational  tests  (OT), 

The  materiel  dovelopor^'Frogrom  Kanagor  is  responsible  for  the  DT  portion 
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of  the  CTP  arid  its  overall  criordination,  aM  pw^ic^itiori  whU.e  OTSii  is 
responsiHe  for  the  OT  portion  of  the  CW  (1^:?4)« 

Durlns  devclopraorit  portion  of  the  CTP,  OTEA  coordinates 

witli  the  natoriel  developer/^i  and  harraonises  the  OT  \dth  the  DT  portion 
of  tho  CIP,  Coordination  i/itii  the  user  is  accomplished  to  determine  user= 
support  required  to  accomplish  OT*  .‘my  changes  vrhich  occur  in  plans  for 
DT  or  OT  are  closely  coordinated  bot*.;ccn  the  materiel  dcveloper/Program 
Kanager  and  03SA,  OTEA  concurrence  in  the  CTP  for  major  ^sterns  is 

r 

required  heforc  the  CTP  can  bp  publidicd  (l|:F4)»  | 

9f  I 

/ill  0Tc2J  conducted  ty  OTSA  is  independent  of  DT.  The  identification  | 

of  operational  issues,  the  preparation  of  the  data  collection  plan  to 
answer  operational  issues,  tlic  preparation  of  the  OT  design  plan,  the  [ 

test  report  and  the  independent  cvaliiatldn  arc  all  accomplished  hy  OTSA 

i 

independent  of  tJie  developor^'^rpgra'a  "Manager  (1JJ:17)  • 

i/honever  possible,  OT  is  conducted  separate  from  DT.  Koi:ever,  in 
some  cases,  DT  and  OT  are  combined  due  to  Imitations  on  ttie  number  of 
prototypes  available.  The  degree  to  Tdiich  OT  is  coabined  td.th  DT  can  f 

vaxy  from  sharing  the  prototype  idth  Ihe  developer  to  merely  observing  t 

DT  to  obtain  answers  to  linited  operational  issues.  The  combination  of 
early  DT  and  OT  is  acceptable  provided  OTil  actively  participates  in  the  i 

tests  and  obtains  the  nocossaiy  OT  infoinatiwi  to  make  an  independent 
evaluation  of  the  test  results  (8:4}* 

Operational  Testing  (OT) 

OT  is  conducted  as  ncces^ay  and  as  early  as  praoticablo  in  tho 
dovelopiaont  pwgram,  beginning  with  early  prototypes  and  continuing  through 
production. 
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■”0T  will  be  accomplished;  ly  user  and  support  personnel 
of  lype  and  qualifications  of  those  exacted  to  use  und 
maintain  the  ^'stea  \dien  depleted  (8:3)," 

OT  will  normally  be  conducted  in  three  phases,  each  keyed  to  the 
appropriate  decision  point,  0T3;\  as  the  Army's  independent  test  agen<y 
is  responsible  for  ass^ing  tl*t  .dequate  OT  is  conducted  for  all  major 
systems,  OTBA  will  actively  participate  with  tte  designated  user  in 
planning  for  and  conduct  of  OT,  and  \/iH  prepare  an  jjidependeht  evaluation 
of  the  adequacy  of  the  testing  and  the  validity  of  the  re salty  of  OT 
(13:H4). 

OT  I is  conducted  to  provide  early  information  as  to  system 
operational  suitability  in  order  to  assist  in  determining  whether  the 
system  should  enter  J\^-Scale  Development,  OT  I may  also  help  identity 
or  refine  critical  istyes  to  be  ex^ined  in  subsequent  operational  tosttyg. 
In  selected  cases  DT  i and  OT  I may  be  combined  ;(i3t*‘J4)* 

OT  li  is  conducted  prior  to  the  production  decision.  It  provides 
an  assesi^ent  of  system  operational  effectiveness  and  suitability, 

IXiring  OT  II,  the  system  is  subjected  to  as  realistic  an  operational 
environment  as  possible  using  small  troop  units  typical  of  those  expected 
to  eventually  be  equiptyd  with  the  i^stom,  OT  II  corresponds  to  lOTdB 
phase  specified  in  DOD  Directive  5000,3  {l3sH4),. 

OT  III  is  accomplished  using  early  production  models  and  has  the 
fundamental  jwrpose  of  assuring  that  the  system  is  operationally  suitaliLe; 
that  all  operationally  critical  issues  have  been  resolved;  and  that  all 
benefits  and  burdens  of  the  system  are  identified.  Earlier  estimates 
ore  validated  and  organisation  and  doctrine  concepts  are  refined  as  well 
as  training  and  logistic  requirements  (l3iH4) • 
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Figure  1 siL^arises  OT^l’s  participation  in  tte  OT  process  and 
indicates  those  agencies  with  which  0T3A  interfaces  in  accoaplishing  its 
mission. 

user  may  conduct  other  operational  tests  ut  any  time  during 
the  materiel  life  cycle  to  reassess  operational  suitabili-fy  or  operational 
effectiveness  of  a ^stea. 

Test  Directorate  for  OT 

pie  designated  user  will  normally  provide  the  Test  Director  and 
test  troops  for  conduct  of  OT  and  OTSfs.  will  provide  the  Deputy  Test 
Director  and  a Test  Cell.  TRilDOC  \/ill  provide  a Deputy  Test  Director 
for  doctrine  and  training  eiqieriiso.  The  Test  Directorate  will  be  filled 
out  with  personnel  from  the  installation  at  which  the  test  is  being 
conducted.  The  OTS.l  Test  Cell  will  train  data  collectors  provided  by 
the  local  commander  at  the  test  installation. 

Following  the  test,  the  Test  Directorate  will  assemble  the  data, 
prepare  a tost  report  and  provide  copies  to  OTEA,  the  user,  and  othe 
participants  in  the  natcriel  acquisition  process^  An  independent  ©•  ?ua- 
tion,  considering  the  test  report  and  all  available  data  will  be  pr-i'pv  ‘'cd 
by  OTSA*s  Evaltiation  Division,  assisted  by  other  divisions  as  necet  s^a?j. 
In  accordance  witti  the  guidance  contained  in  previously  nehtioned 
LOI  of  23  August  1972,  this  independent  evaluation  will  not  include  sn 
operational  suitability  statement,  but  will  assess  the  military  utility 
and  operational  effectiveness  of  the  tested  ^stem.  The  independent 
evaluation  will  then  bo  provided  to  the  ASARC  members  and  the  ASARC  will 
make  the  suitability  determination  (12: 20). 
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OTBii  PARTICIPATIOII  IR  OPEIUTIONiUi  TSSTIIIG 


' . .^1;  r.;-."  J.'ia-. -'.Jiv "ig  ;'u-~ 


Conflict  hetueen  OT  and  DT 

Due  to  failure  of  the  Army  to  publish  an  updated  AR  70-10,  "Test 
and  Evaluation  During  Development  ?jid  Acquisition  of  Materiel,"  there 
continues  to  be  some  conflict  betvieen  DT  and  OT,  The  conflict  is- 
centered  around  service-use  phase  of  DT  which  is  similar  to  OT.  The 
recently  pub3,ishod  Report  of  the  Army  Materiel  Acquisition  Revievf 
Committee  (/iI’IARC)  has  recommended  that  the  service-use  phase  of  DT  be 
discarded.  The  MARC  also  recommended  that  DA  affirm  the  concepts  of 
DT  and  OT  to  emphasize  the  technical  orientation  of  DT  and  the  operational 
orientation  of  OT  and  publish  this  guidance  as  soon  as  possible  (4;v-ll), 
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SU!'S-liuiY  jUJD  CCilf^.ySIONS 


Substantial  changes  have  occurred  in  the  Army's  organization  for 
the  conduct  of  0T£E  since  the  issuance  of  the  Blue  Ribbon  Defense  Panel 
Report, 

The  Army  has  established  a strong  centralized  chain  of  command  for 
OTiS,  that  leads  directly  to  the  Chief  of  Staff  of  the  Army,  v/hile  ACSi’OR 
is  currently  in  this  chain,  upon  its  abolishment  reporting  v;ill  bo  direct 
to  the  Chief  of  Staff. 

The  U,  S.  Army  Operational  Test  and  Evaluation  Agency  (OTEA)  has 
been  established  as  an  agency  independent  of  the  developer  and  the  user 
to  insure  adequate  OT^iE  is  conducted  for  major  systems.  Extensive 
coordination  between  the  developer/Program  Manager,  the  user,  the  Army 
Staff  and  OTEA  occurs  in  the  development  of  the  CTP  for  a major  system 
and  this  coordination  continues  as  changes  to-  the  CTP  occur  to  insure  i 

that  0T<SE  and  DT&S  are  effectively  planned  and  conducted, 

f 

OTEA,  upon  completion  of  each  OT  phase,  reports  the  results  of  its  ; 

I 

independent  test  and  evaJaiation  to  the  Chief  of  Staff  to  assist  the  - 

ASARC/DSARC  in  their  decision  making  process,  ! 

The  author  concludes  that  the  Army  has  established  the  necessary  i 

framework  and  organization  to  implement  -Uie  policy  and  guidance  of  DOD  \ 

f 

Directive  5000,3  with  respect  to  OT&B  for  major  systems.  It  vrill  take  | 

time  for  a true  assessment  of  the  effectiveness  of  OTEA’s  efforts  uith 
respect  to  improving  the  materiel  acquisition  process. 

The  Army  shoiild  take  immediate  steps  to  publish  AR  70-10,  "Test 
and  Evsiluation  During  Development  and  Acquisition  of  Materiel,"  to 
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provide  all  lurmy  agencies  with  policies  and  procedures  for  test  and 
evEJ.uation.  The  author  realizes  that  publication  may  not  be  possible 
until  after  the  planned  reorganization  of  the  Army  Staff  is  complete; 
ho'./ever,  imblication  should  occur  as  soon  as  possible  thereafter.  Hope- 
fully, this  regulation  will  resolve  the  current  conflict  regarding  the 
service-use  aspects  of  DT  v.'hich  involve  operational  type  testing. 

The  author,  based  on  the  research  for  this  study,  would  recommend 
as  a possible  future  ISP  topic,  a study  on  the  role  of  the  U.  S,  Army 
Test  and  Evaluation  Command  in  the  test  and  evaluation  of  major  systems. 
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